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engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
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We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
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We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 
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350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 
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Scott  Cole 

Russel  Wolff 

Abstract 

The  use  of  open  architecture  (OA)  systems  to  guide  acquisition  of  naval  systems  and  the 
“opening  up”  of  proprietary  systems  is  presumed  to  have  produced  significant  cost  savings. 
However,  their  use  may  have  also  introduced  new  forms  of  risk  and  uncertainty  for  the 
acquisition  manager.  Addressing  this  problem,  several  qualitative  research  studies  were 
conducted  to  identify  benefits,  risks,  and  best  practices  from  historical  case  data  involving 
OA,  service-oriented  architecture  (SOA),  and  modular  open  systems  approach  (MOSA) 
implementations. 

Introduction 

The  use  of  open  architecture  (OA)  systems  to  guide  the  acquisition  of  naval  systems 
and  the  “opening  up”  of  proprietary  systems  is  presumed  to  have  produced  significant  cost 
savings.  However,  their  use  may  have  also  introduced  new  forms  of  risk  and  uncertainty  for 
the  acquisition  manager.  Addressing  this  problem,  several  qualitative  research  studies  were 
conducted  to  identify  benefits,  risks,  and  best  practices  from  historical  case  data  involving 
OA,  service-oriented  architecture  (SOA),  and  modular  open  systems  approach  (MOSA) 
implementations. 

The  first  study  focused  on  cost  savings  from  private-sector  implementations  of  SOA 
in  several  industries.  SOA  has  proven  beneficial  in  the  private  sector,  which  has  derived 
benefits  from  SOA  that  include  cost  savings,  agility,  and  flexibility.  Because  SOA  and  OA 
share  comparable  concepts,  the  Department  of  Defense  (DoD)  can  expect  to  realize  some 
of  the  same  benefits  using  OA  as  the  private  sector  gains  from  implementing  SOA.  This 
study  identified  potential  outcomes  and  industry  best  practices  for  the  DoD.  Its  purpose  was 
to  establish  a  benchmark  of  performance  outcomes,  focusing  on  cost  savings  experienced 
in  industry  to  determine  what  the  DoD  can  expect  in  its  push  towards  an  OA  model. 

The  second  study  identified  OA-based  acquisition  risks  and  uncertainties  and 
explored  various  tools  and  techniques  used  by  program  managers  (PMs)  in  successful 
acquisition  programs.  At  the  onset  of  this  study,  it  was  not  clear  how  risk  was  defined, 
perceived,  or  tolerated  at  the  DoD.  Moreover,  the  issue  of  risk  is  a  complicated  problem. 
Unlike  the  private  sector,  the  DoD  does  not  tolerate  or  reward  risk.  OA  introduces  new  risk 
elements  to  DoD  systems  development  and  upgrades;  however,  the  overall  acquisitions 
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approach  is  essentially  designed  to  suppress  risks.  It  is  imperative  to  understand  the  risk- 
suppression  steps  inhibiting  OA’s  potential  ability  to  reduce  costs  and  increase  flexibility. 

The  risk  study  is  augmented  by  primary  research  interviews  with  acquisition  professionals. 

Specific  research  objectives  for  these  studies  included  the  following: 

•  to  examine  relationships  between  OA,  SOA,  and  Navy  OA  (NOA); 

•  to  establish  cost-savings  benchmarks  based  on  industry  performance  for 
traditional  proprietary  architecture  models  and  SOA; 

•  to  identify  the  risks  to  PMs  in  the  Defense  Acquisition  Systems  (DAS) 
ecosystem,  including  various  organizations  involved  with  acquisitions, 
ranging  from  Congress  down  to  a  program’s  risk  project  team,  along  with 
environmental  risks,  consisting  of  rules,  regulations,  laws,  and  customs 
dictating  organizational  behaviors; 

•  to  evaluate  whether  an  OA  strategy  assists  or  hinders  acquisition  programs; 

•  to  ascertain  whether  an  OA  strategy  exposes  a  program  to  unique  risks  and 
uncertainties;  and 

•  to  establish  whether  OA  has  delivered  its  promised  benefits  to  the  DAS. 

Open  Solutions 

The  goals  of  OA  are  to  increase  reuse,  increase  flexibility,  shorten  delivery  time-to- 
market,  reduce  costs,  leverage  competition,  and  improve  interoperability.  Of  these  general 
goals,  decreased  delivery  time  and  reduced  total  ownership  costs  are  the  key  reasons 
behind  the  Navy’s  interest  in  OA.  Open  solutions  offer  new  possibilities  for  solving  business 
problems,  provide  business  interoperability  by  standardization  and  technology  transparency, 
and  decrease  time  to  market  for  key  products  and  services.  Some  general  conclusions 
about  the  current  state  of  OA  are  as  follows: 

•  Organizations  are  adopting  open  technology  platforms  and  open-source 
software  for  critical  business  needs,  and  these  technologies  are  moving  into 
mainstream  business  practices  in  corporations  such  as  IBM,  Google,  Intel, 
and  Pfizer. 

•  The  “open”  movement  has  also  changed  how  society  interacts.  OA,  open 
source,  open  access,  and  open  standards  have  propelled  social  networking 
tools  like  Facebook  and  Twitter  into  astronomical  growth.  Facebook,  Twitter, 
WordPress,  and  Firefox  are  all  built  on  flexible  platforms  that  enable  co¬ 
development  and  co-creation  to  varying  degrees  and  invite  user  opinions. 

•  SOA  and  OA  are  similar  concepts,  and  fulfill  the  requirements  set  forth  by  the 
DoD  for  the  open  systems  initiative,  as  seen  in  Table  1 . 
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Table  1.  Comparison  of  Open  Systems  to  OA  and  SOA 


Open  System 
Characteristics 

Open  Architecture  Characteristics 

Service-Oriented 

Architecture 

Characteristics 

Heavy  emphasis  on 
modularity 

Modular  design  and  design  disclosure 

Modular  services 

Lower  total  ownership  cost 
and  systems  with  longer  life 
expectancy 

Life  cycle  affordability 

Lifetime  costs  decreased  by 
reliability  and  modifiability 
attributes 

Easier,  quicker,  and  less 
expensive  expansion  and 
upgrading 

Easily  upgradable  systems 

Easy  system  upgrades 
through  adaptability, 
extensibility,  and 
modifiability 

High  degree  of  portability, 
connectivity, 
interoperability,  and 
scalability 

Core  concepts  of  scalability  and 
portability,  and  stated  goal  of 
interoperability 

Quality  attributes  of 
scalability  and 
interoperability 

Faster  and  less  costly 
technology  transfer 

Goal  to  optimize  system  performance 

Quality  attribute  of 
performance 

Reusable  application  software 

Reusable  services 

Interoperable  joint  warfighting 
applications  and  secure  information 
exchange  (common  services  and 
information  assurance) 

Quality  attributes  of  usability 
(common  services)  and 
security 

Note.  This  table  was  adapted  from  a  similar  table  in  Azani  (2001). 


Potential  Benefits 

Quantifying  the  tangible  economic  benefits  of  SOA  is  not  an  easy  task.  Return  on 
investment  (ROI)  for  SOA  is  oftentimes  difficult  to  calculate  because  attributes  such  as 
efficiency  are  not  easy  to  quantify.  However,  calculating  ROI  is  critical,  given  that  most 
businesses  look  for  tangible  ROI  when  evaluating  or  approving  new  or  continuing 
investments.  One  British  study  found  that  89%  of  companies  use  “intuition”  or  “guesswork” 
to  calculate  the  ROI  of  their  IT  investments  (DiMare,  2009,  p.  5).  According  to  ZapThink 
Research  (as  cited  in  Schmelzer,  2005),  “Only  by  understanding  the  full  range  of  SOA  value 
propositions  can  companies  begin  to  get  a  handle  on  calculating  the  ROI  of  SOA”  (para.  2). 
Furthermore,  Gartner  analyst  Randy  Heffner  (as  cited  in  McKendrick,  2007)  has  said,  “Any 
attempt  to  assign  a  specific  ROI  to  SOA  should  be  viewed  with  heavy  skepticism”  (para.  3). 
McKendrick  (2007)  further  argued  that  SOA  is  a  set  of  best  practices  that  are  relatively 
intangible.  Some  argue  that  not  only  should  monetary  values  define  ROI,  but  that  ROI 
should  be  defined  by  return  on  closing  capability  gaps  that  are  targeted  by  SOA 
implementation  and  by  nonmonetary  valuations,  such  as  customer  satisfaction  and 
avoidance  of  loss  of  life  (Buck,  Das,  &  Hanf,  2008).  Figure  1  displays  some  nonmonetary 
considerations  for  analyzing  ROI. 
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Figure  1.  ROI  Analysis  Considerations  for  SOA 

(Buck  et  al.,  2008,  p.  13) 

Although  it  is  difficult  to  quantify,  case  studies  have  shown  that  SOA  is  beneficial  in 
the  private  sector,  deriving  benefits  such  as  cost  savings,  agility,  and  flexibility.  Similar 
benefits  could  be  achieved  with  OA  at  the  DoD.  Table  2  displays  some  of  the  tangible 
financial  benefits. 
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Table  2.  Baseline  Data — ROI  Reported  by  18  Selected  Companies  According  to 

Case  Study  Reports1 


Company 

ROI 

Benefit 

(discounted) 

Investment 

(discounted) 

NPV 

Discount  % 

Discount 

Period 

(Years) 

Payback 

(months) 

Blue  Cross  Blue  Shield 

of  KC 

332% 

14,330,000 

3,320,000 

11,010,000 

12% 

6 

20 

Mobile  Telecom 

625% 

10,120,000 

1,400,000 

8,720,000 

12% 

3 

5.6 

Real  Time  Services 

215% 

180,000 

57,000 

120,779 

5 

0 

Global  Provider  for 

Info  Mgmt  Sys 

470% 

8,080,525 

1,417,846 

6,662,679 

12% 

3 

2.5 

Services  and  Fac  Mgmt 

Co 

360% 

2,744,982 

596,674 

2,148,309 

12% 

3 

4.6 

European  based 
telecom 

212% 

5,472,842 

1,753,242 

3,719,600 

12% 

3 

9 

International  Finance 

Firm 

252% 

$6,627,447 

$1,882,568 

$4,744,879 

12% 

3 

6.7 

Healthcare  Provider 

356% 

$13,475,631 

$2,952,633 

$10,522,889 

12% 

6 

6.7 

Global  Media 
Consulting  Firm 

244% 

$1,541,718 

$447,938 

$1,093,780 

12% 

3 

8.2 

Healthcare  Services 

Provider 

346% 

$15,800,000 

$3,500,000 

$12,300,000 

12% 

3 

4.8 

Global  Financial 

Services  Firm 

472% 

$37,140,000 

$6,490,000 

$30,650,000 

12% 

3 

3.9 

Carphone 

42% 

$1,254,000 

$812,000 

3 

30.6 

Johnson  Controls 

81% 

$370,000 

$143,547 

3 

12 

Bank  of  India 

234% 

$23,000,000 

5 

24 

More  Direct 

428% 

$445,395 

$47,270 

$332,251 

5 

5 

International 

Insurance  Provider 

256% 

$1,428,180 

$401,607 

$1,026,573 

12% 

3 

8 

Global  Consumer 

Products  Co 

265% 

$1,118,547 

$306,370 

$812,176 

3 

5.8 

Quicken  Loans 

298% 

$183,000 

Average 

305% 

9.4 

Note.  This  table  was  constructed  using  information  from  the  following  case  studies:  Case  Study 
Forum  (2009a,  2009b),  IDC  Business  Value  Spotlight  (2009a,  2009b,  2009c,  2009d,  2009e,  2010a, 
2010b,  2010c),  IDC  ExpertROI®  Spotlight  (2010a,  2010b,  2010c,  201  Od),  “Shopping  for  SOA”  (2006), 
Nucleus  Research  (2007,  2008),  and  Thoughtware  Worldwide  (2010). 

Researchers  analyzed  34  case  studies,  18  of  which  provided  an  overall  ROI.  Ten  of 
these  cases  were  broken  down  into  various  cost  components.  Based  on  the  case  studies, 
conclusions  regarding  benefits  to  industry  and  best  practices  were  developed.  As  shown  in 
Table  2,  the  overall  ROI  from  industry  SOA  implementation  was  305%,  while  the  ROI  from 
cost  savings  and  cost  avoidance  was  72%. 


1  ROI  =  (Benefit  -  Cost  of  lnvestment)/Cost  of  Investment 
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Table  2  displays  information  from  the  18  case  studies  reporting  overall  ROI.2  ROI 
was  calculated  over  a  three-  to  six-year  period.  All  companies  calculated  a  net  present  value 
(NPV)  with  a  discount  rate  of  12%.  Furthermore,  a  payback  period  was  calculated  for  most 
case  studies.  ROI  was  calculated  with  a  process  of  measuring  benefits,  calculating  total 
investment,  and  then  projecting  the  investment  and  benefit  over  the  time  period  designated.3 

Although  the  industry  achieved  a  72%  cost  savings,  the  DoD  must  weigh  other 
factors  before  it  implements  an  SOA  project.  Benefits  such  as  productivity  improvements 
and  non-quantifiable  benefits  should  also  be  considered,  along  with  factors  such  as 
flexibility,  scalability,  and  reusability,  which  all  allow  for  long-term  improvement. 

Risk  and  Uncertainty 

The  goal  of  project  risk  management  is  to  increase  the  probability  and  impact  of 
positive  events  and  decrease  the  probability  and  impact  of  negative  events  in  the  project. 
This  is  done  through  a  series  of  six  processes  of  project  risk  management:  plan  risk 
management,  identify  risks,  perform  qualitative  risk  analysis,  perform  quantitative  risk 
analysis,  plan  risk  responses,  and  monitor  and  control  risks  (Project  Management  Institute 
[PMI],  2008).  In  practice,  different  PMs  approach  each  step  differently  and  some  may  not 
give  equal  weight  to  quantitative  risk  analysis  as  opposed  to  a  qualitative  approach.  These 
six  processes  will  be  valid  for  most  programs,  especially  in  the  DoD,  as  can  be  seen  in 
Figure  2,  which  shows  the  DoD  risk  management  process,  along  with  the  six  processes  of 
project  risk  management. 


Risk 
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DoD  Risk  M  a  nagement  Process 


1.  Plan  Risk 
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Risk  Track ing 
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Figure  2.  DoD  Risk  Management  Processes  With  Six  Processes  of  Project  Risk 

Management 

(DoD,  2006,  p.  45) 


2  Methods  for  calculating  ROI  varied  because  the  case  studies  were  conducted  by  different  companies. 

3  The  reports  did  not  provide  details  on  exactly  how  benefits  were  measured. 
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•  The  DoD  anticipated  that  OA  principles  would  enable  small,  innovative 
businesses  to  enter  the  defense  market.  The  open  business  model  “was 
envisioned  to  encourage  competition  at  all  system  levels,  therefore  enabling 
small  companies — who  cannot  compete  with  the  likes  of  the  large  contractors 
for  big  Navy  contracts — to  compete  their  solutions  at  the  sub-system  or 
component  level”  (Computerworld,  2007). 

•  Greater  competition  was  expected  to  provide  small-medium  sized  enterprises 
(SMEs)  opportunities  to  enter  the  defense  arena  and  end  eras  of  stove-piped 
systems  and  the  oligopoly  of  defense  contractors  who  provide  expensive, 
monolithic  systems  that  do  not  interoperate. 

•  SMEs,  however,  cannot  participate  in  the  defense  arena  because  of  risk- 
suppression  mechanisms  and  the  exorbitant  costs  to  enter  the  market.  SMEs 
cannot  afford  to  follow  the  bureaucratic  rules  and  restrictions  that  the  current 
acquisition  processes  in  the  DoD  impose. 

•  Systematic  risk  restrictions  at  the  DoD  and  in  the  acquisitions  process  have 
resulted  in  programs  still  going  over  budgets  and  schedules,  despite  attempts 
to  control  budget  and  schedule  risk.  Until  risk  issues  are  addressed,  the  DoD 
will  never  achieve  true  portfolio  management,  nor  will  it  ever  fully  implement 
OA. 

•  The  Risk  Management  Guide  for  DoD  Acquisition  (DoD,  2006)  does  not 
prescribe  specific  methods  or  tools  and  only  provides  general  guidance  and 
accepted  practices  for  DoD  acquisition  professionals  to  follow. 

•  DAS  does  not  present  PMs  with  different  types  of  risk  at  different  stages  of 
their  careers.  Beyond  cost,  schedule,  and  performance  risks,  there  is  no 
formal  recognition  of  other  risk  types.  Despite  this  lack  of  risk  recognition, 
budget  and  program  risk  are  of  constant  concern  to  PMs. 

•  Critical  risk  areas  cited  in  interviews  conducted  with  acquisition  professionals 
were  budget  uncertainty,  program  risk  and  uncertainty,  and  decreasing 
returns  on  increasing  assets. 

•  Not  unique  to  DAS,  risks  associated  with  misunderstandings  between 
different  functional  areas,  the  “talking  by  each  other”  and  linguistic 
discontinuity,  are  heightened  by  a  lack  of  training  and  contradictory  structural 
goals  between  functional  areas  of  a  program,  such  as  between  a  PM  and  a 
contracting  officer. 

•  DAS  is  highly  structured,  consisting  of  well-defined  requirements  and 
milestones,  so  there  is  little  incentive  for  individual  initiative  in  running 
projects  or  programs  and  little  room  for  personnel  flexibility. 

•  Although  the  DAS’s  bureaucratic  nature  is  not  a  risk,  it  introduces  or  amplifies 
risk  as  many  PMs  develop  a  fatalistic  attitude  towards  risk  (i.e.,  “We  have  to 
play  with  the  hand  we  are  dealt.”),  delaying  ramifications  for  incorrect  or  even 
illegal  decisions  that  fall  on  the  program  long  after  the  original  participants 
have  transferred. 

•  OA  has  delivered  cost  savings  and  allowed  faster  system  development  in 
certain  cases;  it  has  also  increased  complexity  and  risk  for  programs. 

NOA  Benefits  and  Lessons  Learned 

With  an  open  systems  approach,  the  Navy  has  derived  a  number  of  benefits, 
including  decreasing  time  to  field  and  upgrading  in-service  systems  faster,  and  modifying  or 
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changing  capabilities  per  fleet  at  reduced  cost.  More  specifically,  OA  resulted  in  benefits  to 
the  Acoustic  Rapid  COTS  Insertion  (A-RCI)  process  and  the  Navy’s  E-2  program. 

Acoustic  Rapid  COTS  Insertion  (A-RCI)  Process 

The  submarine  fleet  saved  $4  billion  while  increasing  sensor  performance  seven-fold 
(Computerworld,  2007).  OA  also  allows  submarines  to  upgrade  software  every  year  and 
hardware  every  two  years.  This  approach  has  been  transferred  successfully  to  other 
submarine  systems  as  well  as  collaborative  efforts  of  the  cross-domain  anti-submarine 
warfare  (ASW)  community. 

A-RCI  followed  an  innovative  approach  in  order  to  leverage  the  benefits  of 
collaboration  between  contractors,  both  small  and  large,  academic  laboratories,  and 
government  organizations.  Lockheed  Martin  served  as  the  prime  contractor  for  A-RCI,  but 
the  focus  was  changed  to  be  a  “prime  system  integrator.”  Even  though  Lockheed  Martin 
would  play  the  major  role  in  the  contract,  the  door  was  opened  to  smaller  contractors  and 
other  organizations  that  usually  could  not  or  would  not  participate  in  the  acquisition  process. 
The  main  vector  for  input  from  small  contractors  and  nontraditional  entities  into  A-RCI  was 
the  peer  review  process  that  selected  between  different  alternatives  and  chose  the  best 
solution,  usually  after  testing  with  real-world  data  (Boudreau,  2006). 

This  strategy  arose  out  of  one  of  the  founding  guiding  principles  of  the  program  in 
that  “no  single  organization  [had]  the  whole  story.”  The  peer  review  process  was  conducted 
under  the  oversight  of  a  Navy  PM  with  the  goal  of  preventing  the  usual  tendency  for  the 
prime  contractor  to  mold  the  program  in  the  most  profitable  direction  for  it,  possibly  ignoring 
competitor’s  solutions  that  may  have  been  more  suitable.  The  peer  group  structures  were 
designed  for  flexibility,  and  an  extensive  set  of  working  groups  were  set  up  to  cover  most 
aspects  of  the  program,  including  a  tactical  integration  advisory  group;  groups  for  specific 
subsystems,  such  as  the  APB-1/2  towed  array;  and,  perhaps  most  important,  an  operator 
feedback  group.  The  composition  of  the  groups  was  fluid  over  the  project  life  cycle  with 
groups  merging  or  even  disbanding  depending  on  the  circumstances  (Boudreau,  2006). 
Figure  3  represents  the  A-RCI  process. 
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Figure  3.  A-RCI  System  Development  Model 

(Barron,  2006,  p.  54) 
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Navy’s  Air  Domain 

The  E-2  program  transitioned  to  a  commercial  computing  plant  with  a  modular 
software  design  through  OA.  Acquisition  cycle  time  was  reduced  from  seven  years  to  2.5 
years,  and  costs  were  reduced  from  over  $200  million  to  under  $1 1  million  (Computerworld, 
2007). 

One  of  the  lessons  learned  from  NOA  was  overcoming  acquisition  and  defense 
cultures.  According  to  a  2007  Computerworld  case  analysis,  the  greatest  obstacles  cited 
were  the  naval  acquisition  and  defense  industry  cultures.  Greatest  resistance  to  NOA  came 
from  those  who  did  not  understand  the  OA  concept,  who  did  not  think  it  would  work,  or  who 
were  not  comfortable  with  change.  The  most  important  obstacles  to  overcome  were  cultural 
issues: 


•  “Not  Invented  Here”  Syndrome.  Navy  personnel  are  resistant  to  being  told  by 
outsiders  how  to  conduct  their  business.  Program  staff  have  generally  been 
working  within  their  programs  for  many  years  and  are  confident  they  know 
how  best  to  continue  conducting  their  program’s  business.  Past  contractors 
who  are  now  employed  by  these  programs  also  resist  change.  This  insular 
environment  limits  the  potential  for  new  ideas  and  increases  resistance  to 
changes  introduced  from  the  outside. 

•  Complacency.  Large  defense  industry  companies  were  content  with  business 
as  usual  because  they  were  making  huge  profits  for  shareholders.  The 
companies  that  develop,  build,  and  upgrade  the  Navy  and  Marine  Corps’ 
National  Security  Systems  (NSS)  had  no  incentive  to  change  their  business 
model  while  they  were  so  profitable.  The  predominant  industry  players 
needed  to  be  convinced  that  profits  would  falter  if  they  did  not  start  producing 
OA  systems. 

•  Lack  of  Asset  Sharing.  This  stems  from  the  “not-invented-here”  syndrome.  It 
is  the  Naval  Enterprise’s  reluctance  to  share  assets  among  domains  and 
programs.  In  addition,  this  internal  attitude  is  the  defense  industry’s 
propensity  for  building  new  systems  from  scratch  rather  than  reusing  assets 
that  the  government  already  owns  and  that  provide  the  needed  capability 
(Computerworld,  2007). 

Recommendations 

Focus  on  Overall  System  Value 

The  overall  value  offered  by  an  open  system  should  be  considered  and  not  only  the 
cost  savings.  Benefits  such  as  flexibility,  scalability,  and  reusability  position  the  DoD  to 
rapidly  adjust  systems  to  changing  combat  missions  and  environments  while  reducing  future 
risk.  The  DoD  should  consider  reducing  the  weight  given  to  ROI  as  a  result  of  cost  savings 
in  its  decision-making  process  and  attempt  to  incorporate  all  associated  benefits. 

Use  an  Incremental  Implementation  Approach 

SOA  is  not  a  one-size-fits-all  solution.  The  DoD  should  adopt  an  incremental 
approach,  implementing  OA  where  results  will  be  immediate.  It  should  assess  current  DoD 
architecture  to  focus  efforts  on  particular  needs  and  requirements.  The  DoD  should  start 
small  with  near-term  or  easily  implemented  requirements,  initially  attacking  the  low-hanging 
fruit  by  introducing  the  SOA  services  that  provide  the  most  bang  for  the  buck. 
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Provide  Adequate  Resources 

Continue  building  the  DoD  infrastructure  to  ensure  any  new  initiatives  are  sufficiently 
resourced.  SHARE  is  a  warning  on  lack  of  resources,  particularly  the  lack  of  personnel  and 
time,  which  ensures  contractors  meet  all  administrative  requirements  concerning  intellectual 
property  rights.  Implementations  that  do  not  have  supporting  infrastructure  and  proper 
resources  could  become  costly  disasters. 

Provide  Greater  Initiative 

DAS  has  evolved  into  a  system  that  concentrates  on  performance  issues,  even  at 
the  expense  of  costs  and  schedule.  Delivery  of  world-class  systems  to  operating  forces 
should  always  remain  a  priority;  however,  the  DoD  should  consider  allowing  PMs  more 
flexibility  in  running  programs.  A-RCI  showed  how  taking  initial  performance  risks,  security 
risks  that  use  a  COTS  strategy,  and  potential  cost  and  schedule  risks  that  use  a  spiral 
development  strategy  could  lead  to  program  success. 

Continue  Accountability 

Many  PMs  inherit  programs  that  have  achieved  initial  success  but  at  the  expense  of 
future  risks  and  stability.  If  a  program  is  of  such  a  length  that  a  PM  has  transferred  before 
improprieties  or  poor  decisions  are  uncovered,  then  he  should  remain  accountable  for 
decisions  in  older  programs  if  the  PM  is  still  in  government  service. 

Support  Greater  Flexibility 

With  a  more  flexible  systems  development  approach,  talent  outside  DAS  could  be 
tapped.  Allowing  for  security  concerns,  modern  problem-solving  methods  such  as  Topcoder 
could  be  used.  The  former  helps  solve  technical  problems  for  the  pharmaceutical, 
biotechnology,  consumer  goods  and  high  technology  industries  with  cash  prizes.  The  latter 
allows  the  programming  community  to  compete  and  collaborate  on  problems  with  contests 
where  members  compete  for  money  and  skill  ratings. 

Implement  New  Metrics 

Due  to  the  innovativeness  of  implementing  an  open  business  model  in  DoD 
acquisitions,  new  metrics  must  be  implemented.  There  are  many  methods  available; 
however,  the  DoD  should  consider  implementing  metrics  to  measure  the  new  economy 
based  on  intellectual  capital  and  knowledge  assets. 

Conduct  a  New  Study 

Conduct  a  study  to  determine  which  DAS  areas  would  benefit  from  OA  and  which 
programs  would  be  hindered  by  OA  and  SOA. 
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Research  Objectives 

examine  relationships  between  OA,  SOA,  and  Navy  OA  (NOA); 

establish  cost-savings  benchmarks  based  on  industry  performance 
for  traditional  proprietary  architecture  models  and  SOA; 

identify  the  risks  to  PMs  in  the  Defense  Acquisition  Systems  (DAS) 
ecosystem,  including  various  organizations  involved  with 
acquisitions 

evaluate  if  an  OA  strategy  assists  or  hinders  acquisition  programs; 

ascertain  if  an  OA  strategy  exposes  a  program  to  unique  risks  and 
uncertainties;  and 

establish  if  OA  has  delivered  its  promised  benefits  to  the  DAS. 
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Context:  The  “Open”  Movement 

Defense  Acquisitions  wants  better  products  for  less 


•  Open  Solutions  deliver  costs  savings  and  capability  improvements  to  the  private  sector 

•  Open  Business  systems  may  deliver  these  same  benefits 

•  anticipated  that  OA  principles  would  enable  small,  innovative  businesses  to  enter  the 
defense  market. 


Service-Oriented  Architecture 


•  (SOA):  “approach  for  sharing  functions  and  applications  across  an  organization  by 
designing  them  as  discrete,  reusable,  business-oriented  services.”  (GAO,  2005) 


Open  solutions 


•  offer  new  possibilities  for  solving  business  problems,  provide  business  interoperability  by 
standardization  and  technology  transparency,  and  decrease  time  to  market  for  key  products 
and  services. 
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Comparison  of  OA  and  SOA 


Open  Architecture  Characteristics 

Modular  design  and  design  disclosure 
Life-cycle  affordability 

Easily  upgradable  systems 

Core  concepts  of  scalability  and  portability,  and  stated 
goal  of  interoperability 

Goal  to  optimize  system  performance 
Reusable  application  software 

Interoperable  joint  warfighting  applications  and  secure 
information  exchange  (common  services  and  information 
assurance) 


Service-Oriented  Architecture 
Characteristics 

Services  are  modular 

Reliability  and  modifiability  attributes  decrease 
cost  over  the  lifetime  of  the  system. 

Adaptability,  extensibility,  and  modifiability 
provide  ease  of  upgrading  a  system. 

Quality  attributes  of  scalability  and 
interoperability 

Quality  attribute  of  performance 
Reusable  services 

Quality  attributes  of  usability  (common  services) 
and  security 
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Potential  Quantitative  Benefits 


•  Case  Studies  Analyzed  the  ROI  of  SOA  implementation  in  18  different 
companies-  336%  ROI  with  72%  ROI  from  Cost  Savings 


Company 

Reported  ROI 

Calcuated 

ROI  from  Cost 
Savings  /Cost 
Avoidance 

Average 

Annual  Cost 
Savings 

Average 

Annual  Cost 

Avoidance 

Average 

Annual 

Productivity 

Improvement 

Benefit 

(discounted) 

Investment 

(discounted) 

NPV 

Discount  % 

Discount 

Period 

(Years) 

Payback 

(months) 

Blue  Cross 

Blue  Shield 

332% 

330% 

$2,380,000 

$0 

$90,000 

$14,330,000 

$3,320,000 

$11,010,000 

12% 

6 

16.7 

Mobile 

Telecom 

625% 

136% 

$1,100,000 

$0 

$3,570,000 

$10,120,000 

$1,400,000 

$8,720,000 

12% 

3 

15.3 

Global 

Provider  for 

470% 

-18% 

$0 

$387,853 

$2,827,485 

$8,080,525 

$1,417,846 

$6,662,679 

12% 

3 

43.9 

Services  and 
Fac  Mgmt  Co 

360% 

-100% 

$0 

$0 

$1,140,000 

$2,744,982 

$596,674 

$2,148,309 

12% 

3 

European 

based 

212% 

-18% 

$478,463 

$0 

$1,801,860 

$5,472,842 

$1,753,242 

$3,719,600 

12% 

3 

44.0 

International 

Finance  Firm 

252% 

-31% 

$101,015 

$329,054 

$2,669,439 

$6,627,447 

$1,882,568 

$4,744,879 

12% 

3 

52.5 

Global  Media 
Consulting 

244% 

107% 

$111,609 

$198,140 

$332,626 

$1,541,718 

$447,938 

$1,093,780 

12% 

3 

17.4 

International 

Insurance 

256% 

7% 

$143,839 

$0 

$427,328 

$1,428,180 

$401,607 

$1,026,573 

12% 

3 

33.5 

Healthcare 

Services 

346% 

146% 

$0 

$2,870,000 

$3,720,000 

$15,800,000 

$3,500,000 

$12,300,000 

12% 

3 

14.6 

Global 

Consumer 

265% 

165% 

$270,689 

$0 

$195,366 

$1,118,547 

$306,370 

$812,176 

12% 

3 

13.6 

Average 

336% 

72% 

27.9 
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Potential  Qualitative  Benefits  as  Seen  in  the  Commercial  World 


Benefit  Categories 

Examples  of  Qualitative 
Measurements 

Relationship  to  the  DoD 

Business  Staff  Efficiency 

Information  delivered  to 
managers  more  quickly  and 
accurately  improves  decision¬ 
making. 

Timely  and  accurate  delivery 
of  information  vital  to  military 
leaders. 

Business  Credibility 

Equates  to  more  business 
because  other  companies 
view  their  system  as  available 
and  reliable. 

Productivity  improvement 
through  availability  and 
reliability  of  systems  used  by 
the  DoD. 

Reduced  Duplication  of  Effort 

Information  entered  once, 
available  to  all  users. 

Ensures  accuracy  and 
consistency  of  data.  It  also 
saves  time  inputting  data  or 
fixing  mismatched  data. 

Faster  Time-to- Market 

Difference  in  the  amount  of 
time  a  product  is  available 
compared  to  the  current  time 
to  market. 

Faster  delivery  of  vital 
intelligence  or  logistics  when 
and  where  required. 

Scalability 

The  ability  to  increase  size  or 
volume  without  degradation. 

Scaling  of  service  in 
accordance  with  changing 
mission. 

Flexibility 

Flexibility  is  achieved  through 
increased  agility  and  potential 
for  reuse. 

Flexibility  allows  for  quick 
adaptation  to  environmental 
changes. 
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Small  Business  Participation 

DoD  anticipated  greater  Small-Business  participation  through  OA 

•  Small  Business  Innovative  Research  (SBIR)  Contracts 

Results  showed  actual  Small  Business  participation  sub-optimal 


•  Contract  Funds  Awarded  from  1  February  2012-  31  March  2012 


$9,133,000,234 

Award  Total 

$53,409,358 

Average  Contract  Awarded 

$537,112,301 

Small  Business  Award  Total  minus  contracts  over 

$100  million 

$8,346,994 

Average  Small  Business  Contract  Awarded  minus 
contracts  over  $100  million 

5.88% 

Percentage  of  Awards  given  to  Small  Businesses 
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Types  of  Risk 

When  interviewed,  program  managers  identified  the 
following  types  of  risk  they  have  to  mitigate: 


Risk 

Number  of 

Times 

Mentioned 

Risk 

Number  of 

Times 

Mentioned 

Risk 

Number  of 

Times 

Mentioned 

TECHNICAL 

10 

PERFORMANCE 

6 

ENVIRONMENT 

SAFETY  & 

OCCUPATIONAL 

HEALTH 

5 

COST 

5 

SCHEDULE 

4 

MANUFACTURING 

3 

OPERATIONAL 

2 

INTEGRATION 

2 

ENTERPRISE 

ARCHITECTURE 

2 

SYSTEM  LEVEL 

2 

MATERIAL 

1 

PROGRAM 

1 

5/18/2012 


Risk  and  Uncertainty 


Determined  that  Risk 
Suppression  excluded 
Small  Companies  from 
participation  in  an  Open 
Business  Model 


•  Heavy  bureaucracy  makes  it 
difficult  for  small  businesses  to 
compete. 


•  Because  of  these  risk 

suppression  measures,  entry 
costs  prohibit  entry  of  small 
businesses  into  the  acquisitions 
environment. 


DoD  Risk  Management  Process 


1.  Plan  Risk 
Managamant 

2.  Ida ntify  Risks 


\ 


Risk  Analysis 


5.  tonitcrand 
Control  Risks 


3.  Pa  rfo  rm  Qua  I  tat  rva  Analysis 

4.  Pa  rform  Quant  Hat  iva  Analysis 


£.  Plan  RiskRasponsas 


Risk  Mitigation 
Planning 


5.  tonltarand 
Control  Risks 


Risk  Mitigation 
Plan 

Implementation 
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Top  Internal  Barriers  to  OA 


Current  flow  of  money  -  Budget 
allocations  through  organization 

Lack  of  defined  reference 
architecture 

Lack  of  OA  understanding 

Lack  of  resources 

Unwillingness  to  break  existing 
contracts 

Lack  of  OA  education  and  training 

Lack  of  formal  requirements 

Perceived  up-front  investment  too 
expensive 


•  Lack  of  incentives 

•  Lack  of  contractual  requirements 

•  T&E  and  Certification  issues 

•  Fear  of  Change  -  ranked  #1  by 
respondents  w/A-RCI  experience 

•  Lack  of  centralized  configuration 
management  control 

•  Lack  of  leadership  support 

•  Current  Organizational  Structure 

•  Lack  of  accountability 


A-RCI  Success  Story 

Demonstrated  practical  application  of  Open  Systems 
methodology  by  successfully  implementing  an  improved 
sonar  system  while  beating  budget  and  time  goals: 

•  Life  cycle  costs  were  improved  by  a  factor  of  close  to  5 

•  a  seven-fold  increase  in  processing  capability 

•  mean  operator  success  rate  increased  by  a  factor  of  four 

•  mean  number  of  false  alarms  reduced  by  40% 


detection  and  classification  time  improved  by  27  minutes 
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A-RCI  Development 

Model 


“Build  Test  Build” 


Operational 

- V 

Performance 

Requirements 

Specs 

(The  Fleet) 

(Watfaie  Center) 

|  |  Indmtry 

Gouanmert 


Fleet 

Mary  Pnouktos 

(GHR,  Small  Business, 
Academia,  Industry 


Spiral  Development 


Moore's  Law  & 


\  Tech  Insertion  \ 

(Architect/ 

\  /  reduces  costs  J 

Software  component 
Development /Reuse 

*  Com petitij^e  selection 

*  Integration 
•Test 

(Mary  piv/kiers) 


Prime 
System 
Integration 

(tnctoshy) 


Life 

Cycle 

Support 


COTS  Hardware 

ProcLremert  -  Packaging 
(Mary  prwitiers) 


Gov  Oversight  (Warfare  Centers) 
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A-RCI  Best  Practices  and  Risks 


Best  Practices  and  Successful 

Introduced  Risks 

Strategies 

A-RCI 

COTS 

Security  Risk 

Program  Risk  (bureaucratic  friction) 

Incremental  Strategy 

Operational  Risk  (initial  deployed  system  does  not  meet 

user  requirements) 

Cost  Risk 

Schedule  Risk 

MOSA 

Integration  Risk 

Open  Capabilities-Based  Model 

Integration  Risk 

Program  Risk  (bureaucratic  friction) 

5/18/2012 


Conclusions 

DoD  must  address  risk  management  to  implement  OA/SOA 

•  Regardless  of  risk  suppression  measures,  current  programs  still  miss  time 
and  budget  requirements 

•  DoD  should  examine  the  commercial  world  risk  construct  that  includes  the 
upside  as  well  as  downside  of  risk 

•  Until  the  total  risk  constructs  is  addressed,  the  DoD  will  never  achieve  true 
portfolio  management  nor  will  it  ever  fully  realize  the  potential  of  OA. 

DoD  must  weigh  other  factors  in  addition  to  cost-savings  when  considering 

Open  Systems  approaches 

•  Productivity  Improvements-  measured  by  meaningful,  valid  ROI  projections 
that  need  to  be  established 

•  Flexibility,  Scalability,  and  Reusability-  measured  by  Real  Options  Analysis 
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Recommendations 

Implement  New  Metrics 


•  Traditional  ROI  insufficient  for  the  DoD’s  needs:  Current  ROI 
estimates  rely  on  cost-savings  or  avoidance  data  that  does  not 
provide  a  unique  numerator. 

•  Portfolio  optimization  and  real  options  for  forecasting  future 


Incremental  Implementation  Approach 


•  Attack  the  low-hanging  fruit  by  introducing  SOA  services  that  provide 
the  most  bang  for  the  buck. 


Provide  Greater  Initiative  for  PMs 


•  A-RCI  showed  how  taking  initial  performance  risks,  security  risks 
using  a  COTS  strategy,  and  potential  cost  and  schedule  risks  using  a 
spira  development  strategy  could  lead  to  program  success 
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